SINGULAR SPHERICAL MAXIMAL OPERATORS ON 
A CLASS OF STEP TWO NILPOTENT LIE GROUPS 



Detlef Muller Andreas Seeger 

Abstract. Let = M?" x R be the Heisenberg group and let fit be the normaUzed surface measure for 
the sphere of radius t in R^". Consider the maximal function defined by Mf = supj^Q |/ * /it|. We prove for 
n > 2 that M defines an operator bounded on U'{H") provided that p > 2n/(2n — 1). This improves an earlier 
result by Nevo and Thangavelu, and the range for LP boundedness is optimal. We also extend the result to a 
more general setting of surfaces and to groups satisfying a nondegeneracy condition; these include the groups of 
Heisenberg type. 



1. Introduction 

Let G be a finite-dimensional step two nilpotent group which we may identify with its Lie algebra q by 
the exponential map. We assume that g splits as a direct sum g = tn © 3 so that 

[lr,w]C3, [rt5,3]=-{0}, 

and that dim(rD) — d, dim(3) = m. 

Throughout we shall make the following 

Nondegeneracy Hypothesis. For every nonzero linear functional e 3* the bilinear form 

ro X ro ^ M 
'^'^ '■ {X,Y)^u;i[X,Y]) 



is nondegenerate. 

Note that the skew symmetry of J^^ and the nondegeneracy hypothesis imply that d is even. 
There is a natural dilation structure relative to in and 3, namely for X e ro and [/ G 3 we consider the 
dilations 

6t : iX,U) ^ {tX,t^U). 

With the identification of the Lie algebra with the group St becomes an automorphism of the group. 
In exponential coordinates {x,u), x £ M.'^, u E M'", the group multiplication is given by 

(1.1) (x, u) ■ (jj,v) = (x + y,u + V + x^Jy) 

where x^Jy — {x^Jiy, . . . ,x*Jmy) G and the Ji are skew-symmetric matrices acting on M'* (i.e. J* = 
— Ji). For u G M"* we also form the skew-symmetric matrices Ju = TlT^i^i'^i nondegeneracy 
hypothesis is equivalent with the invertibility of Ju for all u ^ 0. 
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The most prominent examples are the Hcisenbcrg groups which arise when d = 2n, m = 1 and 
J = Ji is the standard symplectic matrix on M^". These belong to the class of Heisenberg-type groups 
(termed H-type groups in [9]), for which = — 4|Mp/, so that the nondegcneracy hypothesis is clearly 
satisfied in this case. Note that in general m has to be small compared to d (see [9] where the connection 
with Radon-Hurwitz numbers is pointed out). The class considered here has been introduced by Metivier 
[10] in his study of analytic hypoellipticity; the nondegcneracy assumption is termed "Condition (H)" in 
[10]. There are many groups which satisfy the nondegeneracy condition but which are not isomorphic to a 
Heiscnberg-type group; we give an example in §7. 

Let E be a smooth convex hypersurface in lu and let be a compactly supported smooth density on 
S. We make the following 

Curvature Hypothesis. The Gaussian curvature of S does not vanish on the support of fj,. 
Define the dilate /Xt by 

(1.2) (Mt,/) = J f{tx,Q)dii{x). 
We recall the definition of convolution 

(1.3) = j f{y,v)g{x - y,u - V + x*Jy)dydv 

and define for Schwartz-functions the maximal operator M by 

Mf{x,u) =sup\f* iJ,t{x,u)\. 



We prove the following sharp result. 

Theorem. Suppose d> 2. Then M extends to a bounded operator on Lp{G) if and only if p > d/{d — 1). 

Remarks, (i) Other more "regular" spherical maximal fimctions on the Heiscnberg group have been con- 
sidered in [2], [15]. In these papers the maximal functions are generated by measures on hypersurfaces and 
the averaging operators are Fourier integral operators associated to local canonical graphs. In our work 
the maximal functions are generated by measures on surfaces of codimension m + 1, and the associated 
canonical relations project with fold singularities. 

(ii) A previous result is due to Nevo and Thangavelu [12] who considered the case of spherical means 
on the noncentral part of the Heisenberg groups (m = 1) and obtained boundedness in the smaller 

range p > {d- l)/(d- 2), d > 2. 

(iii) Our theorem is an analogue of Stein's theorem [16] in the Euclidean case. The necessity of 
the condition p > d/{d — 1) follows from the example in [16]; one tests M on the function given by 
f{y,v) = |j/|^^''(log |y|)^^x(y, w) with a suitable cutoff fimction x- The methods in this paper arc not 
sufficient to establish LP boundedness for p > 2 for the case d = 2 (that is, for an extension of Bourgain's 
result [1] in the Euclidean case). 

(iv) The result should remain true for any nilpotcnt Lie group of step < 2; i.e. the nondegeneracy 
hypothesis should not be necessary. This is currently an open problem. 

(v) As a corollary of the estimate for the maximal operator one obtains the pointwise convergence 
result limt^o A*t * f{x) = cf{x) almost everywhere, if f G LP and c = J dfi. Moreover the L^ bounds of 
the maximal operator are relevant for certain results in ergodic theory, where one needs to have pointwise 
control for large t. 
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(vi) We use in an essential way the invariance of the subspace tr under the dilation group {St}. Namely 
this implies a favorable bound for the principal symbol of {d/dt)fj.t on the fold surface of the associated 
canonical relation. A similar phenomenon was observed in [11] for averages along light rays. 

(vii) One can replace the measure on tr by a measure supported on a perturbed subspace 2B which is 
transversal to the center but no longer invariant under {St}; then the phenomenon in the last remark does 
not occur. In the above coordinates 20 is given as 

(1.4) W = {{x,Ax),x &'^'^}, 
where A = (Ajj) is a m x d matrix. Define a measure by 

(M^/)= J f{tx,t'Kx)diJL{x), 

we also set '■= IJ-i- Consider the maximal operator defined by 

(1.5) mA/ = sup|/*m^|. 

t>o 

For general A we then prove the partial result that is bounded for p > (Sd— l)/(3d— 4). We conjecture 
that boundedness holds for p > d/{d — 1) which by our theorem holds true for A = 0. 

Notation: Given two quantities A and B we write A < B ii there is a positive constant C, such that 
A<CB. 



2. Preliminary decompositions 

We shall present the argument for the maximal operator in (1.5). We shall denote by Aj the j*^ 
column of A and by ||A|| the matrix norm of A with respect to the Euclidean norms on Mf^ and M™. In 
what follows we shall always assume that ||A|| < Ci for some fixed Ci (and various bounds may depend on 
Ci). If ||A|| occurs explicitly in an estimate then we are interested in the behavior for A — > 0, as the case 

of our Theorem corresponds to A = 0. 

We note that by localizations and rotations in M'' one can assume that /i has small support and that 
the projection of S to tt) is given as a graph Xd — r(x'), x' = (xi, . . . ,Xd-i), so that Va;'r(0) = and so 
that /i is supported in a small neighborhood of (0, r(0)) (we may assume that |V:r'r(x')| < C(7^co/100) 
where cq, Cq are defined in (5.10) below). Note that a rotation has the effect of replacing the matrices Ji 
in the group law by Q*JiQ with Q G SO{d). We thus will need to prove an estimate which is uniform in 
these rotations. 

Using the Fourier inversion formula for Dirac measures we may write 

li^{x,u) = x{x,u) jj e'('^(^<'-r(^'))+^-("-^^))dadT 

where X is a smooth compactly supported function and the integral converges in the sense of oscillatory 
integrals (thus in the sense of distributions). 

We split the integrals by introducing dyadic decompositions in (ct, t) and then also in cr, when \a\ < |r|. 

Let ^0 G C^(]R) be an even function so that Co(s) = 1 if \s\ < 1/2 and supp((^o) C (—1, 1). Also define 
Ci(s) = Co(s/2) - Ci(s) and for A; > 1, 1 < Z < fc/3, 



(2.1.1) 


/3o(ct,7-) 




(2.1.2) 


/3fc,o(CT,r) 


= Ci(2-V'^' + M2)(l-Co(2-M) 






-Ci(2-V'^' + |t|2)Ci(2'-V) 


(2.1.3) 




= Ci(2-V^' + k|2)Co(2t'=/^]-'=-V) 
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Then observe that _ 

k>l l<i<fe/3 

and for A; > the function Pk,o is supported where tr « 2*^ and |r| < 2*^, f3k,i is supported where |t| « 2*^ 
and \a\ w 2'="' and Pk is supported where |r| w 2*^ and |cr| < 2^'=/^. 
Define 



(2.2.1) 


K°{x, 


u) 


(2.2.2) 




u) 


(2.2.3) 


k\x, 


u) 



moreover for f > define the dilates 

[ifO, K^'\k^]{x, u) = t-(rf+2™)[;f0^ j^k,i^ K'']{t-^x,t-^u). 

Note that = + Efe>i + Ei<i<fe/3 + ^t')- 

Since iiT^ is a bounded compactly supported function the associated maximal function is controlled 
by the appropriate variant of the Hardy-Littlewood maximal function and therefore ([17]) we have the 
inequality 

\\snp\f*K^\\\^<CM, 

for 1 < p < 00. 

Using known estimates for oscillatory integral operators with fold singularities and additional almost 
orthogonality estimates we shall derive in §3 and §5 the following estimates. 

Proposition 2.1. Suppose k > 0. Then for < I < k/2> 

(2.3) II supl/ * < x/fc2-'=('^-2)/2(l + ||A||2V/2||/||2; 

t 

moreover 

(2.4) II sup 1/ * i^,'=|||2 < ^2-'=(''-2)/2(l + ||A||2'=/3)i/2||/||2 

t 

To obtain V results we shall interpolate with weak type inequalities proved in §6. 
Lemma 2.2. Let fc > 0. For all a> we have 

(2.5) meas{{{x,u) : sup\f * k'^'\x,u)\ > a}) < k2''-'{l + \\A\\2%-'\\f\\, 
forO<l< k/2> and 

(2.6) meas({(x,w):sup|/*if,'=(x,^.)| >a}) <A;22'=/3(l + ||A||2'^/3)a-i/||i. 

t>o 



We interpolate by the real method and obtain 
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Corollary 2.3. Suppose l<p<2 andk>0. Then forO<l < k/3 

(2.7) II sup 1/ * K^'^Wlp < CpA;i/f2-'=('^-i-'^/f)2-'(2/p-i)^^ ^ ||A||2')i/f 

moreover 

(2.8) ||sup|/*^j'=||| < Cpfci/f2-'=('^-4/3-''/f+2/3p)(i + ||A||2'=/3)i/P||/||2. 



Now if p < 2 we may sum in k and I and see that is bounded if d — 4/3 — d/p + l/{3p) > which 
is equivalent to p > (3d — l)/(3rf — 4) (showing the estimate mentioned in remark (vii) in the introduction). 
If A = we get a better bound, namely that boundedness holds if d — 1 — d/p > oy p > d/{d — 1). 
This proves our main Theorem. 

3. Square functions and almost orthogonality 

It is advantageous to introduce cancellation in the above kernels, modulo small acceptable errors. 
Indeed 



jj K^'\x,u)dxdu + jj K'^{x,u)dxdu 



< Cjv2-'=^, 



for all = 0, 1, ... , and this estimate follows by an integration by parts in the {x, u) variables. Thus there 
is a Cq° function b which is equal to 1 on supp(x), and constants 'yk,i, 7fe so that 



(3.1) 



K'''\x,u)dxdu = jkd II b{x,u)dxdu 



J J K^{x,u)dxdu = jj b{x,u)dxdu 



where 
(3.2) 

We define 



hk\ + hk,i\ < Cn2 



-kN 



(3.3.1) 
(3.3.2) 



l&'\x,u) =i^'=''(a;,u) -7fe,;6(x,«) 
KJ^{x,u) = K^{x,u) - 7fe6(a;,u) 



and denote by /C^ ' , /Cj their dilates, as before. Then the functions /C^ ' , /C^ have integral zero. 

Since the maximal operator generated by the kernel b (with nonisotropic dilations) is bounded by the 
nonisotropic Hardy-Littlewood maximal operator we see that for 1 < p < oo 



sup|/*(/Cf''-ii','' 



<C;V,p2-'=^| 



Now in order to deal with the main term we shall use the following standard lemma in the subject 
which is an immediate consequence of a similar one stated in [17, p. 499]. 
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Lemma 3.1. Suppose that 



Then 



,,2\l/2 

sup V F„(-,s) J <Ai 
«e[i,2] ^ 



sup sup <C{Ai + \/a[M). 

n se[l,2] 2 



We omit the proof. Using Lemma 3.1 one sees that the estimates 

II sup 1/ * < v^2-'=('^-2)/2(l + ||A||2')i/2||/||2 

t 

II sup 1/ * ^^=1112 < x/fc2-'=('^-2)/2(l + ||A||2*=/3)i/2||/||2 

t 

foUow from the following estimates which are uniform in s e [1)2]. 

/.^„ ., ,,ox 1/2 

(3.4) - 

(3-5) (E||/ 

n 

for / < A;/3, and 
(3-6) (Ell/ 

n 

(3.7) (E||/ 



(EII/*^24G) <^/fc2-'=('^-l)/^2V2||/||, 

)'^' < Vfc2-'=(''-3)/22-'/2(i + ||A||2' 



, 9 k,i 
dt * . 



t=2"■^ 



rfe ||2^'/' 

* '^2"s II2 



\l/2 ^ 

) <^^2- 



k{d-l)/2+k/6\ 







I dt *J 


4=2" s 



< Vfc2-'=('^-3)/2-'=/6(l + ||A||2'=/3)||/||2. 



Note by scaling that it suffices to prove these estimates for s = 1. We shall ffist use the cancellation 
of the kernels K,2ns ^^'^ ^l^s to show certain almost orthogonality properties (for the sums in n) and then 
we use stronger estimates for oscillatory integrals to establish decay estimates for fixed n. 

An almost orthogonality lemma. We first state a simple and presumably well known consequence of 
the Cotlar-Stein Lemma. 

Lemma 3.2. Suppose < e < 1, A < B/2 and let {T„}^i be a sequence of bounded operators on a 

Hubert space H so that the operator norms satisfy 

(3.8) ||T„|| < A 
and 

(3.9) ||T„T*,|| < B22-^I"-"'I. 
Then for all f e H 

(3.10) (Ell^"/ll') <CA^e-HogiB/Am. 

n=l 
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Proof. For N > 1 consider the operator 

which maps / to the sequence (Ti/, . . . , Tjv/, 0, 0, . . .)• Now UTatII = \\T*Tn\\^/'^ where T*Tn : H ^ H in 
given by 

N 

Tn'^n! = ^ T*Tnf. 
n=l 

We let S„ = T*Tn and observe that 

\\SlSi\\ = \\SuSt\\ = \\T^TkT;Ti\\ 

< \\T;:\\\\TkTi*UTi\\ < mm{A\B^2-^''-'\'}. 

The standard Cotlar-Stein Lemma [17] gives 

oc 

\\T^Tn\\< 5]max{ sup \\SlSi\\'/^ sup WSkStW'/'} 

™_„ k—l=m k—l=m 

m=oo 

and thus 

oc 
m= — oo 

< C^e-^A^logiB/A). 

Thus ||T/v/||^2(jj) is dominated by the right hand side of (3.10), and the assertion follows by taking the 
limit as A'' — > oc. □ 

Remark. We proved Lemma 3.2 by using the statement of the Cotlar-Stein Lemma. Using the proof of the 
Cotlar-Stein Lemma one can also show the following more general fact: If ||T„T*, || < a^{n — n') then 

^ 1/2 / X 1/2 



n=l jGZ 

Of course, Lemma 3.2 is an immediate consequence of this inequality. 

Almost orthogonality estimates. Here we wish to apply Lemma 3.2 to convolutions on groups. If 
Tf = f * g we first note that its adjoint is given by T*f = f * g* where g* = g{-~^)- Moreover using 
Minkowski's inequality and the unimodularity of nilpotent Lie groups one obtains the standard convolution 
inequality 

Il,/*.9||2< ||.9*||l||,/!|2=||.9||l||,/||2. 

We now fix k, I and s € [1, 2] and derive almost orthogonality properties for the operators of convolution 
with )C2"s- 

Notice that forn < the function /Cjng is supported in a (small) ball of radius C2" (in fact in a smaller 
nonisotropic ball). Moreover we have \Vy,vK:^'\y,v)\ < 2'=('"+2) and using the cancellation of /Cjng we 
obtain 

|/C^'' * (/C^;'j*(a;,u)| < 2'=('"+2)2" if n < 0. 
By scaling and applying Schur's Lemma we obtain 

(3.11) \\f*l^o'X * (1^2^)% < 2'=('"+2)2-l"-"'l||/||2 
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first for n < n' and then by taking adjoints also for n < n'. This and the following estimates are uniform 

in s e [1,2]. 

Similarly we get 



(3.12) 

and also 
(3.13) 
(3.14) 

In §5 we shall prove the inequalities 
(3.15) 



die''/, diic'^J, y 



ds 



* s- 



ds 



< 2^(^+4)2-1"-"'! 11 jii^ 



/ * s 



2"'s 



ds 



2 

fc 

2"'s^ 



as 



^ 2'=('"+4)2" 



(3.16) 

for I < k/3, and 

(3.17) 

(3.18) 



QJ^kl 



ds 



<2-fe(d-3)/22-V2(i + ||A||2')||/||2 



||/*if'=||2<2-'=(''-i)/22'=/6||/||2 

5s 2 



< 2-'=(<«-3)/22-'=/6(i + ||A||2'=/3)||/||2. 



By scaling and by (3.2) the same inequalities hold with JsT*^'' and K'^ replaced by /C^' and K,^ and with 
dsK''^\ dsK'' replaced by dsKL^r.,, d^IC^r,,, for 1 < s < 2. 

Now the inequality (3.4) follows from (3.15) and (3.11) if we apply Lemma 3.2 with A = 2-'=('^-i)/22V2 
and B = 2'=('"+'*). Similarly (3.5) follows from (3.16) and (3.12), (3.6) from (3.17) and (3.13), and (3.7) 
from (3.18) and (3.14). 

The next two sections are concerned with the derivation of inequalities (3.15-18). 



4. Preliminaries on oscillatory integral operators with folding canonical relations 

We shall reduce matters to estimates for oscillatory integral operators whose canonical relations have 
two-sided fold singularities. We consider localizations near the fold surface and the estimate goes back to 
Phong and Stein [13] for certain conormal operators in the plane; the general case is implicit in Cuccagna's 
paper [3] . For the version needed here we refer to [6] . 

Let rj <E R" X R" be an open set and let F be an open set in some finite dimensional space. We consider 
phases (p{x,y,^) and amplitudes ax{x,y,j), {x,y,^) G x f2 x F, and assume that 

(4.1) \dy^,p{x,y,^)\<C 

(4.2) \dy^axix, y,j)\< CAd^l+l'^l^/^ 

say, for all multiindices a, (3 with |a|, < lOn, with uniform bounds in O x F; we also assume that all 
derivatives depend continuously on the parameter 7. 

We shall assume that 

dp = {{x,fx,y,-fy)} 
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is a folding canonical relation, i.e. for each point Pq = (a;o,yoj7o) we have 

(4.3) rank(^^^(Po)>n-l, 
and for unit vectors U, V 

(4.4) ^%{Po)V = =^ \{V,Vy)det^%\>c, 

(4.5) C/VL;(^o) = =^ |(C/,V.)det</?^'J >c, 

for some c > 0. 

We consider the oscillatory integral operator Tx[b] defined by 

Tx[b]fix) = j e'^'^^-'y'^^b{x,y,j)f{y)dy 

which is bounded on all if b is bounded and compactly supported. We shall take for b certain localizations 
of the symbol in terms of the size of det ip"y. Let r] be smooth and compactly supported in (—1, 1) so that 
ri{s) = 1 for \s\ < 1/2 and set 

Piix, y, 7) = r?(2' det ^%{x, y, 7)) - r]{2^+^ det ^^ix, y, 7)), 

so that Pi localizes to the set where | dety'^'j^l « 2~K We also define 

Cx{x,y) = l- ^ Pi{x,y) 

2i<Ai/3 

so that I det ip"y\ < A~^/^ on supp(Ca)- 

Then there is a neighborhood U of (.To,?yo,7o) so that for all a\ satisfying (4.2), supported in U the 
following estimates hold for the operator norms: 

(4.6) \\TxM]\\^.^^. < Ci2'/2a-"/^ 2' < A^/^ 
and 

(4-7) ||rA[aACA]|L.^^. < CiAi/'^-"/^. 

These estimates are a consequence of Theorem 2.1 in [6]. 

5. Reduction to oscillatory integral operators 

We now consider the operator of convolution with K^''- and give the proof of the bound (3.15). The op- 
erator dsK'^''' is more singular, but its estimation is rather analogous, so we shall point out the modifications 
needed for (3.16) at the end of this section. The estimations for K'^ and dgK^ will be similar. 

Since K''''' is compactly supported in a fixed neighborhood we may use the translation invariance to 
reduce to the case that / is also compactly supported in a fixed neighborhood of the origin. Thus it suSices 
to show the desired bound for the operator with Schwartz kernel 



(5.1) 



Xi{x, u)K^'\x -y,u-v + x*Jy)x2{y, v), 
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for suitable compactly supported smooth functions Xi and X2- In what follows we set A = 2*^ and then by 
a change of variables the kernel (5.1) can be written as 

(5.2) H^'\x,u,y,v) = >^"'^' JJ e'^*^'''''''^'''''''^\o{x,u,y,v)m{a,T)dadT 



where 



[X 



, u, y, V, (T, t) = (T{xd - yd- r(a;' - y')) + t ■ {u - v + x*Jy - A{x - y)) 



and where |r| 1 and \a\ K, 2"' on the support of rji ; specifically 



r?;(<7,T)=Ci(V'^2 + |T|2)Ci(2'a), 
and xo(a;, u, y, v) = xi {x, u)x{x -y,u-v + x*Jy)x2{y, v). 

Notation. We let P : R"^ — > R'^~^ be the linear map with Pcj = ei, i = 1, . . . ,d — 1 and Pcd = 0. We also 
use the notation P for the {d — 1) x d matrix 

P={I 0) 

and P* for its transpose. 

Stationary phase calculations. We wish to apply stationary phase arguments to reduce matters to the 
estimation of an oscillatory integral operators without frequency variables (see e.g. the general discussion 
in [5]). 

We shall apply a scaled Fourier transform on M^+^j in the {xd,u) variables. Define 



y'x9{x',Xd,u) = J J e-'^^^''''+^-^^g{x',Zd,w)dzddw; 



then (A/27r)(™"'"^)/^.FA is a unitary operator and thus, if Ti^'^ denotes the operator with Schwartz kernel 
H^-^ wc have to prove that TxH^'' maps to itself with operator norm 0(A^('^+'")/^2'/^). Let X3i^d,u) 
denote a smooth compactly supported function which is equal to one whenever jxdl + |w| < 10, and define 
-^A,i by 

J^x,i9{x',Xd,u) = X3{xd,u) J J e-'^^^''''+^-'^^g{x',Zd,w)dzddw; 
moreover let J^a,2 = ^x — Then the Schwartz kernel of J^x^iH^''- is given by 

(5.3) A^+i J e'^*(^'"'^''''^)6;(a;, u, y, v, e)de 

where with 

9 = {zd,w,a,T) 

the phase function \1/ is given by 

*(a;, u, y, v,9)=- XdZd - u ■ w + a{zd ~ yd - ^{x' - y')) 

+ t\w-v + AP* {x' - y') + Ad{zd - yd) + {x'\ Zd)Jy) , 

and the amplitude is given by 

bi{x,u,y,v,e) = X3{xd,u)xo{x' ,Zd,y,w)r]i{a,T). 
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For the error term J^x,2H^'^ wc have a similar formula, only with xs replaced by 1 — xs- Then in 
view of the support properties of (1 — xs) we see that |V0^,u;*| > \xd\ + \u\ on supp(l — xs) and by 
integration by parts with respect to the {zd, w) variables we see that the kernel of !F\^2H^'^ is bounded 
by Cn)^"^~^^~^ i\xd\ + \u\)~^ ■ Moreover this kernel is supported on a set where \xd\ + |u| > 1 and where 
k'l + \y\ + \v\ < C. Thus, with an obvious application of Schur's Lemma we conclude that the operator 
T\^2H^''' is bounded on LP' with operator norm 0(A^^) for any A''. 

We return to the main term T\^iH^'^ and it remains to be shown that 

(5.4) II^A,iH^''|| < 2'/2A-('^+'")/". 

Note that for fixed {x,u,y,v) the phase function is a polynomial of degree < 2 in the 6 variables 
and that the Hessian is nondegenerate. 

Indeed, 



= -Xd + eprV + o- + T*Ad 



(5.5) 



=T-U 



= - z; + {x'\zd)Jy + KP\x' - y') + kd{zd - Vd) 
K = Zd-yd- nx' - y') 




and with S denoting the column vector in with coordinates Sj = Jjj/ + Aid we have 



\I>" — 



Clearly the linear equations "^g — have a unique solution 6'crit = {zd, w, r, ij)crit) with 

izd)ciit{x, u, y, v) =yd + T{x' - y') 

{w,)critix, u,y,v) ^v,- {x'\yd + T{x' - y'))Jiy - e\KP\x' - y') - KdT{x' - y') 
{Ti)ciit{x,u,y,v) = Ui 



a„it{x, u, y,v) = Xd-"^ Ui{e*^Jiy + Aid) 



i=l 



and we can apply the method of stationary phase (with respect to the 2(m + 1) frequency variables 9). 

Setting 

$(x, u, y, v) := *(x, u, y, v, 6'crit(a;, u, y, v)) 
(5.6) 

m 

= -Xd{yd + r{x' - y')) -^Ui{vi - {x'\yd + r{x' - y'))Jiy - Ai^Vix' - y') - e\AP\x' - y')) 

we obtain that 
(5.7) 

/N-l 
^i\^{x,u,v,v,e)^^f^^^ u, y, V, e)de = e'^*(^'"'2''") ^ £j(x, u, y, v)X-^ + R^\x, u, y, v) 

3=0 
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where 

(5.8) £^j{x,u,y,v) = 

{2i)~^ ( det{^00{x, y, u, v, 0crit{x, u, y, v))/2wi) ^{■^'ggDe, DeYhix, u, y, v, 9) 

and 

(5.9) \R)t{x,u,y,v)\ < Civ||6i|U^^,+,„A-^ < C^,2'(™+2+2^)A-^. 

Here we have apphed Lemma 7.7.3 in [7]. 

Since 2' < A^/^ the error term i?^' (which is compactly supported) defines a bounded operator on 
with norm 0(A~^^'"+^+^^/^) which for large N is much better than the desired bound in (5.4). 

Claim 5.1. The operators with kernels A~-'fj(a;, ^,1/, i))e''^*^^'"'^'''^ have operator norm 

Q^)^-{d+m)/2-j/32l/2^ 

This clearly implies (5.4). 

Geometry of the canonical relation. 

Wc consider the canonical relation C$ = {x, u, ^u', y, v, ~^y, —^v) and the singularities of the maps 
PL '■ {y,v) ^ ($x,$u)j PR '■ {x,u) ($y,$„). It is our objective to check the analogues of (4.3-4.5) and 
we will have to verify a few elementary linear algebra facts. 

Let A denote the (rf-l)x((i— 1) matrix T" (a;'-?/') and let B denote the column vector r'(a;'-y') e 
recall that we may assume that |B| is small. Indeed if 



(5.10.1) Co = min || J„ 

(5.10.2) Co= max ||J„ 



-1||-1 



we may assume that 



Now y»L is explicitly given by 



\B\\ < Cq-^co/IOO. 



= -XdT'ix' - y') + PJ„y + T'{x' - yY^JuV + u*AdT'{x' - y') + u'AP' 

^x, = -yd - T{x' - y') 

^u, =-{vi- {x'\yd + T{x' - y'))Jiy - e\kP\x' - y') - KuT{x' - y')). 

We compute the differential Dp^ as 

f{xd-epuy-u*Ad)A + PJuP* + BelJuP' PJuCd O' 
(5-11) I ^* -1 J 

where / is an m x m identity matrix and C is m x (d — 1) matrix with rows C'i = x'^PJiP*^ + ydC^^JiP* — 
{e^iy + A,d)B* + e*AP' + r(a;' - y')e*aJiP* and c is the column in M™ with c, = {x'\0)Jied + e^^y. In 
this calculation the skew symmetry of the Jj is used. 

We now compute the determinant of (5.11) and obtain 
(5.12) det $(;,„),(y,„) = (-1)'' det ({xd - e'^J^y - u*Ad)A + PJ^P* + E{B)) 
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where 

(5.13) E{B) = Be^JuP* + PJuCdBK 

Here we used the factorization 

aA + PJuP* + BeyuP' PJubA ^ f ctA + PJ^P* + E{B) PJucA ( I 
B* -1 J \ -1 J \ -B^ 1 

Note that E{B) is a skew-symmetric (d — 1) x {d — 1) matrix and so is PJuP^ + E{B). Thus, since 
d — 1 is odd, the rank of PJ^P^ + E[B) is at most d—2, and the following lemma shows that for small B 
the rank is equal to d — 2. 

Lemma 5.2. Suppose that 

" " - 4Co 

Then the following holds: 

(i) If We Ker (PJ„P* + E{B)) then 

(5.14) le^uP'Wl > 'f\\W\\. 

(ii) dim Ker (PJ„P* + E{B)) = 1. 

(Hi) If X belongs to the orthogonal complement of Ker (PJ„P* + E{B)) then 

(5.15) ||(PJ„P*+£;(P))X||>|||X||. 

Proof. Observe that 

\\Em\ < 2Co\\B\\. 
Thus if W e Ker (PJ„P* + E{B)) and \\W\\ = 1 then 

1 = \\P'W\\ < \\J-m\J^P'W\\ 

< \\Ju'\\{KJuP'W\ + \\PJuP'W\\) = ||J-i||(|ey„P*W| + \\E{B)W\\) 
<Co^(|e*J„P*W|+2Co||B||) 

and thus, if < cq/ACq we obtain |ey„P*W^| > co/2 which is (5.14). 

Let Su = Ju + E{B). Since Su is skew symmetric, it can be diagonalizcd over C, and the eigenvalues 
are imaginary. The bounds (5.10.1/2) are still vaUd if J^^ is acting as a linear transformation on C''. Let 
?7 e be a unit eigenvector of 5„ so that SuT] = iXr] and ||?7|| = 1; then 

|A| = WS^vW > WJuVW - mBM > CO - \\E{B)\\ > CO - 2Co||i3|| > | 

by assumption on B. Hence |A| > co/2 for every eigenvalue iX of Su- In particular Su is nondegenerate. 
But then PSuP* = PJuP* + E{B) has rank d—2 and therefore a one-dimensional kernel and all nontrivial 
eigenvalues of Su are also eigenvalues of PS'mP*. This implies for vectors X orthogonal to the kernel of 
P5„P* that 



PSuP*X>^\\X\\ 



which is (5.15). 
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Lemma 5.3. Let 'Qi he a symmetric positiv definite matrix on M" and let S be a skew- symmetric matrix 
onR". Then: 

(i) For all u ^ 0, the matrix uSJl + S is invertible and the inverse satisfies the bounds 

(5.16) \\{a^A + S)-^<\a\-^\^-% 

(a) If S is invertible then aQl + S is invertible for all a and we have the bound 

(5.17) ||(a2l + S)-'\\ < 2\\S-^ zf \a\ < (2||2t||||5-i||)-\ 

Proof. 

For a unit vector e in M" we get 

||(c72l + 5)e|| > |((a2l + ^)e,e)| = |(a2te,e)| > 

Here we have used that by the skew symmetry of S we have {Se,e) = 0, and also that ||2t~^|| = l/Amin, 
where Amin is a minimal eigenvalue of 21. This establishes invertibility and the bound (5.16). 

If in addition S is invertible and a is small we may simply use the Neumann series to get invertibility 
ofCTa + S*. Namely, if |cr| < (2112111115-111)"^ we get (o-2t + S')-i = S'-i(/ + X;°li(-l)V^'(2lS'-i)^') and the 
bound (5.17) is immediate. □ 

Lemma 5.4. Let d > I be an odd integer, let f2i be the cone of real symmetric positive definite £ x i 
matrices and let be the set of all skew symmetric £ x £ matrices with rank £ — 1. 

For S G fl^ choose a unit vector es in the kernel of S and let ws be the orthogonal projection to the 

orthogonal com,plement of es- 

Then for A e fii, S e ^2, cr e M we have 

(5.18) det{aA + S) = a{Aes, es) det(7rs(£7^ + S)-k*s) + ct^F(A, S, a) 
where F is a smooth function on Oi x O2 x R. 

Proof. Let Q = Q{S) be an orthogonal transformation with e^Q = (0, . . . , 1). Then 

e'M+s)«=(<'^«;«» 

where Sq is a skew symmetric invertible {£ — 1) x {£ — 1) matrix, Aq is positive definite, a G and 
T] = {Aes, es). We apply Lemma 5.3 to aAo + Sq and factor 

/ aAo + So <Ta\ _ / I 0\ / aAo + So aa 

\ aa* at]) ~ \aa\aAo + So)~'^ l)\ ar] - a'^a*{aAo + So)~'^a 

and conclude that 

det(c7^ + S) = det(c7^o + So){ar] - a'^a\aAo + S'o)"^a). 
The assertion follows since det{aAo + So) = det{'jTs{aA + S)'jTg). □ 

We now proceed to verify the conditions (4.3-5) in §4. By Lemma 5.3 the determinant of ^'('^ 
can only vanish when a := a^^ = — e^J„y — u*Aj; vanishes. In this case the dimension of the kernel 
^(x u) (y v) equal to the dimension of the kernel of PJ^P* + E[B) with B = Y'{x' — y'), thus equal to 1. 
Thus rank (^('a;,„),(y,^)) > d + m-1 everywhere. 
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In order to verify (4.4) let Vl be a nonvanishing vector field which is in the kernel of Dpi, when the 
mixed Hessian (5.11) becomes singular (i.e. when Xd — e^^Juy — u*Ad = 0). Then 

and with A = T"{x' - y'), we have ql = B^Wl and 

(5.20) {a A + PJ„P* + Be'aJuP' + PJ„edB*)VFi = 0; 

moreover the functions hL,i are in the ideal generated by the Wl,] (and the coefficients can be computed 
from (5.11)). To get a nontrivial kernel (when a = 0) we must choose a nonvanishing vector Wl satisfying 

(5.20) . Notice that then |e^J„P*Wi:,| is bounded below, by (5.14). By Lemma 5.4 we have 

V,:(det$'(; (^ ,)) = {-lfFi{x,y,u)e'dJuP*WL+F2{x,y,u,v){xA - ey„2/ - «*Ad) 
where Fi and F^ are smooth and Fi does not vanish. Thus |^L(det [y v))\ — '-'^ 

Next wc consider the map pn and let Vr be a nonvanishing vector field which is in the kernel of Dpn 
(or the cokernel of (5.11)) when Xd — e^uj/ — u*Ad = 0. Then 

where by (5.11) the functions ha^i vanish when Xd — e^^JuV — u'-Kd = and 

W'ji [a A + PJuP' + Be'dJuP'] + 9rB' = 
W'jiPJued - 5fl = 0; 

thus since A is symmetric and Ju skew symmetric we have essentially the same equation for Wl above, 
except that J„ is replaced by — Ju'- 

(5.21) {a A - PJuP' - PJuCdB* - e'juP')WR = 0. 

Moreover qr = c^^JuP^Wr does not vanish by (5.14). As Xd — e^^Juy — u*Ad does not depend on x' we get 
VR(det (j,,^)) = Fi {x, y, u)e\JuP^WR + ^ix, y, u, v){xd - e^J^y - u^Ad) 

with smooth functions Fi, F2 and nonvanishing Fi. Thus | V/{(det {y v))\ bounded below on the 

zero set of det and we have verified the statements analogous to (4.3-5). 

Proof of Claim 5.1, conclusion. For small I the bound is immediate from Hormander's standard 
estimate for nondegenerate oscillatory integrals ([8], cf. (5.12) and Lemma 5.3 above). For large / we can, 
by Lemma 5.4, rewrite the amplitude f j as a finite sum 

e]{x,y,u,v) = 2'^' J2 Ci{2'+'det^l^^^^^^^)qi+i{x,u,y,v) 

\i\<C 

where the qi+i are compactly supported and smooth and satisfy the estimates d"y^^ vQi+i — 0(2'"). Since 
2' < A^/'^ this type of blowup is covered by (4.2) and wc can apply the estimate (4.6) and see that the 
operator with kernel A has operator norm < 22j'A-JA-('^+")/22V2. This implies our claim. 
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Modifications for the proof of (3.16). By scaling we need to consider tlie operator of convolution with 

Let (p be as in (5.2) and 

d ,x u y V . 
p{x ,Xd,u,y,v,a,T) = — .^(-, ^, -, ^, a, r) 

(5.22) 

m m 

= (T{-Xd + yd + {x' - y') ■ V^'T{x' - y')) + 2 ^ Ti{-Ui + Vi- x'Jiy) + ^ Tie\k{y - x). 

i=l i=l 

As before we set A = 2^^ and observe that our operator is a sum of an operator 5'^'' with Schwartz 
kernel 

G^'\x, u, y, v) = A"+2 J J eiV(-.».!/.''.-.-)p(a;', Xd, u, y, v, a, t)xo{x, u, y, v)mi'J, T)dadT 

and an operator which has similar properties as i?^'' above (thus satisfies estimates which are better than 
claimed in (3.16)). 

We now need to carry out the stationary phase calculations as before for the kernel J^\,iQ^''' (since the 
contribution from J^x.2Q^''' is again negligible). It has the form of (5.3), except that bi is replaced by Ac; 
where C( is given by 

ci{x, u, y, V, 9) = bi{x, u, y, v, Zd, w, a, t)p{x', Zd, w, y, v, a, t). 
Then by stationary phase the Schwartz kernel of J'x^iQ^ '' can be expanded as 
(5.23) 

yrn+2 / e'^^C™'''^)^ (x, y, i;, 0)d0 = e*^*^^'"*'') ^ f j (x, w, y, u)A1-^' + i^^^ 

where again the error term i?^' is easy to handle for large A'' and £^ is defined as in (5.8) but with bj 

replaced by Cj. 

In order to finish the proof of (3.16) it is now sufficient to establish that the operator T^'^ with kernel 
^^l-j^^^x^(x,u,y,v) satisfies thc bound 

(5.24) ||7;.^''||i2^i2 < Ai-('^+")/22-'/2(l + j|A||2'). 

The differentiation in s causes a blowup by not more than A and by our previous analysis it follows that 

(5.25) \\r^''\\L-^L- < 2'/2Ai-('^+")/2(22iA-i)J-. 

If j = 1,2,... this estimate is suflacient for (5.24) since then 2'/^(2^'A~^)-' < 2~'/^ by our restriction 
2' < AV3. 

This crude estimate does not suffice for the leading term in the asymptotic expansion when ||A|| is 
small (or zero). 

However note that when A = the coefficient of in (5.22) vanishes on the critical set where 9 = 
6„it{x, u, y, v) since d^/dr = on that set. We get 

p{x',Zd,crit,Wcrit,y,v,acTit,Tcrit) ={xd - e^d^uV " «*Ad)((x' - y') ■ Va:'r(x' - y') - V(x' - y')) 

rn 

+ 2Y,Ui{e\KP\x' - y') + e*Adr(x' - y')). 

i=l 

Since \xd — e^J^y — u*Ad| « 2^' on thc support of q and since the coefficients of are 0(||A||) we now 
gain an additional factor of 0(2~' + ||A||) in the estimate (5.25) for j = and thus establish (5.24) also for 
3 = 0. 
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Modifications for the proof of (3.17), (3.18). The only reason for the modified definition (2.2.3) 
(replacing (2.2.2) for I > fc/3) is the preservation of the symbol estimates (4.2), needed for the validity of 
(4.6), (4.7). The estimation for K'^ is exactly analogous to the estimation of K'''^ when I < k/3, and the 
same statement applies to the s-derivatives. Only notational modifications are needed. 



6. Weak type (1,1) estimates 

We are now proving the weak type inequality (2.5). The proof of (2.6) is omitted since it is exactly 
analogous. 

We apply standard Calderon-Zygmund arguments (with respect to nonisotropic families of balls on 
nilpotent Lie groups, see [4], [17]). Cf. also [14] and related papers on singular Radon transforms. 

Let 

Bs = {ix,u) : \x\<S,\u\<5^} 

and denote by Bg its complement. 

Since we have already checked the bounds for the maximal function it suffices to check the following 
Hormander type condition for Z/°°(1R+) valued kernels: 

sup sup / sup\K^'^{iy,v)-\x,u)) -K^''ix,u)\dxdu<k2''-\l + \\A\\2^) 

S>0 {v,v)eBs J B^^g t>o 

which follows from the two estimates 

/■ lA-M/, ,-1, ^Mr Mww</ 2'=-'(l + ||A||20, 

Indeed we use the first bound for the 0{k) terms with 2~^'=(™+^) < 2~"5 < 22'=(™+^) and the second bound 
for the remaining terms. We then sum the series in n. Using scaling we see that the latter estimates are 
equivalent to 



r r ok 

(6.1) sup / sup \K^''-{x - y,u- V + x^Jy) - K^'''{x,u)\dxdu < 
{y,v)eBr Jbi^^ se[l,2] 



2'=-'(l + |lA||2'), 
2fc(m+2) niin{r-\r}. 



Because of the support properties of the kernel the integral on the left hand side is zero if r ^ 1. Now 
assume that r < 1. Since \V K'^'''{x,u)\ < 2'=('"+2) the bound 2'^("'+^V in (6.1) is immediate. It remains to 
show that 

II sup |i^,^''|||^<2'=-'(l + ||A||2'), 

s6[l,2l 

and this follows from 

(6-2) ||ii''=''||^ < 1, 

(6.3) ||5«i^,^''||^ <2'=-'(l + ||A||2'). 

By an integration by parts in cr, r we see that 

(6.4) \K''^\x,u)\<Cn 



N 



(l + 2fc-'|a;d-r(a;')|)^ (1 + 2'=|u - Aa;|) 

from which (6.2) immediately follows. Moreover from (5.22) one obtains by the same argument 
\dsK^'\x,u)\ is bounded by C^2'^^'(l + ||A||2') times the right hand side of (6.4). Consequently we 
obtain (6.3). This finishes the proof of the weak type inequality (2.5). □ 
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7. Appendix 



In this section we give the example of a two-step nilpotent Lie group G, with 10- dimensional Lie 
algebra, which satisfies the nondegeneracy condition but which is not isomorphic to a group of Heisenberg 
type. 

For /X = (/xi,/X2) e let 



E,., 



and define the 8x8 matrix 

then 
(7.1) 

Let Q be the Lie algebra which is 



/Ml 








-/U2 




/"I 


















\ 







Ml 







E, 




-{ 


-El 








J, 



dct = (/i* + 



as a vector space, with Lie bracket 
[X + U,Y + V]^0+ J(i,o)l", ^*^(oa)i"). 

By (7.1) the group identified with q satisfies our nondegeneracy condition. We now prove by contradiction 
that is not isomorphic to a Heisenberg-type Lie algebra. 

Assume; that there is a Lie algebra isomorphism a : g ^ g where g is a Heisenberg-type algebra. Then 
= fD ® 3 where 3 is the center and a is a linear isomorphism from 3 to R^. 

Now with respect to orthonormal bases ui, . . . , Ms on tt) and ug, uio on 3 and ei, . . . , eg on and eg, eio 
on 'M? the map a is given by the 10 x 10 matrix 

'A 
L B 

where A is an invertible 8x8 matrix and B an invertible 2x2 matrix. 

Now let X = Yl^^iXiUi, Y = and express a; € 3* in terms of the dual basis as w = 

wiUg + W2uIq. Then, since g is of Heisenberg type we have co{[X, ¥]) = x^ J^y with = —{w\ + W2)I; in 
particular 

(7.2) \AeiJ^\ = {wl+wlf. 

Now if w = a^ji (thus B^-fx = (^1,^2)*) then 

x^Jb^^V = uj{[X,Y]) = {a')-^u{a[X,Y]) = [aX,aY]) = (Ax)*J^(Ay) 



so that A^J^A = Jstn and therefore 



det Jstfj. = (det A)'^ det 



Thus by (7.1) and (7.2) we obtain = {detAf{nf + and therefore, if (a, 6) and (c,d) are the 

rows of the matrix | Aet A\~^/^B^ , 

lA + lA = ((a^i + b^J^,2f + (c/ii + diJ,2)^)^, 

for all /X e ]R2. Thus 

Ml + M2 = ((a^ + c^)Mi + (^'^ + d'^)lA + 2(a^ + cd)niii2))'^ 

for all /X e M^. This implies + = b"^ + = 1 and setting p = ab + cd we obtain after a little algebra 
that 

(4p2 + 2)^1^2 +4p(/i2+/i2) =0 

for all /U e M^. This implies both 2p'^ + 1 = and p = 0, thus a contradiction. □ 
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